Skip to content

feat(cli-vector): analyse mbtiles BM-1270 #3444

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 34 commits into
base: master
Choose a base branch
from
Open

Conversation

ccbblin
Copy link
Contributor

@ccbblin ccbblin commented May 12, 2025

Motivation

analyse cli to generate some information from mbtiles

Modifications

add analyse cli

Verification

check report created

@ccbblin ccbblin marked this pull request as ready for review May 23, 2025 03:38
@ccbblin ccbblin requested a review from a team as a code owner May 23, 2025 03:38
@ccbblin ccbblin requested review from Wentao-Kuang and tawera-manaena and removed request for a team May 23, 2025 03:38
@ccbblin ccbblin force-pushed the feat/etl-analyse branch from ee368ab to 9051405 Compare June 2, 2025 23:16
@ccbblin ccbblin added the container build pull request container for testing label Jun 3, 2025
@ccbblin ccbblin force-pushed the feat/etl-analyse branch from bc43c7f to 8c26085 Compare June 3, 2025 03:39
@ccbblin ccbblin added container build pull request container for testing and removed container build pull request container for testing labels Jun 3, 2025
Base automatically changed from feat/etl-create to master June 4, 2025 02:12
Wentao-Kuang and others added 17 commits June 4, 2025 14:13
… properties BM-1268 (#3440)

1. This pull request extends the [feat/etl-create] branch, not the
[master] branch.
2. The changed files contain short-circuiting code to enable faster
debugging that needs to be purged before approval.

---

### Motivation

We are currently migrating our Vector ETL system into basemaps as a new
`cli-vector` package. We are also re-building the system in the process.
As a part of the re-build, we have streamlined the system's architecture
in the following four steps:

1. `extract`
2. `create`
3. `join`
4. `analyse`

This work implements an extension to the base branch's implementation of
the `create` CLI command. This work introduces a series of functions for
handling dataset features of specific Shortbread layers for which we
require special tagging.

### Modifications

| Diagram |
| - |
| ![][diagram1] |

1. Implements a switch-style pattern for identifying the Shortbread
layer to which a feature is assigned.
2. Implements bespoke functions for overriding a feature's metadata and
properties for the following Shortbread layers:
   - `contours`
- `place_labels` - This one is weird. We will need to discuss the
approach for this one.
- `pois` - This one doesn't actually do anything as it never receives
any features that will meet its conditions for special tagging. We can
discuss this also.
   - `public_transport`
- `street_labels` - This one's logic is very similar to the `streets`
logic.
- `streets` - This one's logic is very similar to the `street_labels`
logic.
   - `water_polygons`

#### Processing a Vector Stac Item

The system processes a Stac Item file created via the **cli-vector**
package's new `extract` command like so:

1. Parse the file as a runtime `VectorStacItem` type.
2. Download the source file (i.e. a GeoPackage file).
3. Convert the source file into an NDJSON file.
4. Parse each line (i.e. feature) of the NDJSON file, generalise it
(i.e. simplify, add/remove attributes, or discard).
5. Process the collection of generalised features into an mbtiles file.
6. Upload a copy of the mbtiles file to the same directory as the Stac
Item file.
7. Update the Stac Item file's contents and overwrite it.

| Diagram |
| - |
| ![][diagram2] |

### Verification

1. Established test cases for the special tagging functions for the
following Shortbread layers:
   - `contours`

More to come if desired.

<!-- external links -->

[feat/etl-create]: https://github.com/linz/basemaps/tree/feat/etl-create
[master]: https://github.com/linz/basemaps/tree/master

[diagram1]:
https://github.com/user-attachments/assets/6a405417-59fb-4127-8630-cb0901c12618
[diagram2]:
https://github.com/user-attachments/assets/2994bd5d-c668-4b99-957e-57623da7c6a1

---------

Co-authored-by: Wentao Kuang <[email protected]>
### Motivation

Join cli to join all mbtiles and create stac

### Modifications

### Verification

check cotar and stac files correctly created
@ccbblin ccbblin force-pushed the feat/etl-analyse branch from fccb528 to acdce6e Compare June 4, 2025 02:19
@ccbblin ccbblin added container build pull request container for testing and removed container build pull request container for testing labels Jun 4, 2025
@ccbblin ccbblin added container build pull request container for testing and removed container build pull request container for testing labels Jun 5, 2025
@ccbblin ccbblin added container build pull request container for testing and removed container build pull request container for testing labels Jun 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
container build pull request container for testing
Development

Successfully merging this pull request may close these issues.

3 participants